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Abstract — The focus of this article is on designing a decision 
support system, which optimizes maintenance and operation 
processes in petrochemical industries based on Markov decision 
process. Since a consistent process is located at the end of 
processes range, these industries’ machines form a continuous 
chain, and the resulting flow on them always continue from the 
feed to the final product. Flaws result in equipment failure, 
hiatus in production, and even decrease of product quality and 
in practice any case of production stopping costs a lot to 
resume. 

This research has been conducted to analyze the effects of 
changes resulting from high demand of market on equipment. 
The new formula calculates relations between maintenance and 
repair controls clearly with regard to the weight of lack of 
production in case of equipment failure in different production 
conditions in a continuous system based on Markov decision 
process. This model optimizes a set of maintenance controls of 
different production conditions in each period, and in this way 
optimized values of condition, control, and optimized policy of 
control are obtained. This model has been performed in 12 to 72 
month periods. Having chosen the optimized policy of control 
and obtaining the sample number of surveyed statistical 
population, the indicators have been measured for six months 
both before and after performing the optimized policy of 
control where the overall effectiveness of system increased to 87 
from 82.7, and positive and acceptable changes were seen. 

Index Terms — Decision Support System, Maintenance, 
Markov Decision Approach, Optimization, Overall Systems 
Effectiveness 

I. INTRODUCTION 

In the third millennium, different companies and 
organizations face numerous challenges to have a successful 
presence in the business world, and these challenges are 
sometimes conflicting. Meanwhile, one of the most 
fundamental factors of changing the return on asset is the 
Equipment plays an undeniable role in qualities of products, 
and a production of high quality is one of the major objectives 
of any system [2]. We consider a petrochemical factory with a 
high number of equipment, devices, and machines which all 
contribute to the production process. Since planning for 
conducting maintenance operation of equipment is done 
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periodically, production process is affected, and even could 
stop during conduction [3], [4]. Otherwise, sudden failure of 
equipment may cut production systems. Sometimes, High rate 
of production because of high demand, leads to imposing 
more pressure on production systems which itself expedite the 
failure of systems [5]. The motivation for doing so is resulted 
from the need for development of optimized policies of 
maintenance, and operation for time-continuous systems, so 
that a correct and accurate method could be designed for 
supporting the production [6]. This can be possible by 
developing the concept of effectiveness using the Markov 
decision process according to the weight of lack of production 
in case of equipment failure in continuous systems [7] - [1 1]. 
Overall equipment efficiency (OEE) in Total Productive 
Maintenance (TPM) can be defined based on Tkachima's 
Definition (1998) as following [12]: 

OEE = availability x production ratex quality rate (1). 
Therefore, based on the given definition in (1), and because of 
the studied equipment being hidden, Overall System 
Effectiveness (OSE) is the integrated availability, the 
capability of production, and quality measurement of all 

equipment calculated by (2): OSE=n^ =i 0 EE^ (2) 

The rest of this article is organized as following. The subject 
has been reviewed in the first two sections. In the third 
section, the issue has been presented in detail and the problem 
is expressed by theoretical backgrounds of Markov decision 
process, considering the lack of production in case of 
equipment failure in continuous systems, and a mathematical 
model to maximize the Overall System Effectiveness. In 
section four, the calculation method based on the defined 
process in section three has been offered. Also, a numerical 
example of application is shown, and in section six the 
conclusion of this article is done. 

II. REVIEW OF SUB JECT 

In this article, support system is designed to optimize 
maintenance and operation processes in petrochemical 
industries based on Markov decision process, considering the 
lack production in case of equipment failure. The final 
decision is, then, taken by calculating operating costs 
depending on different states along which the operation time, 
risk due to the equipment age and irregular inspection, and 
replacements are considered depending on time and operating 
costs. In another research by Koochaki, Javid et al. (2013) 
about the impacts of maintenance based on equipment 
conditions for workforce planning and maintenance [13], the 
maintenance policy is used to predict a failure event 
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according to the circumstances of each part, and therefore, it 
tries to prevent any unwanted failure, and unnecessary 
maintenance and repair activities from happening. The two 
factors of Condition Based Monitoring (CBD) or Age Based 
Replace (ABR) have been considered both in parallel and 
series multi-component systems regardless of staff limitation 
and with internal staff of the unit, or with external 
maintenance and repair staff with their attendance time. 
AlDurgam, Mohammad et al. (2012), carried out a study titled 
common policies of optimal operation, maintenance and 
repairs to maximize overall equipment efficiency [14], and 
performed it in one of the refinery units. This model optimizes 
maintenance and repair operation, and production rate for 
each period. The decision maker (controller) can choose the 
optimal policy using the space vector of system state (belief 
state). Zhang Zaifang et al (2010) carried out a research on 
conceptual design of production and maintenance. They 
investigated the increasing importance of services such as 
maintenance to a manufactured product, and their role in 
improvement of customer satisfaction and promotion of 
consistent consumption [15]. Another research was done by 
Murat Kurt et al (2010) about the optimal maintenance of a 
critical Markov system with incomplete limited repairs. They 
considered a problem in a critical periodically inspected 
system using time-discrete Markov process with limitation on 
number of repairs before replacements to achieve the optimal 
maintenance [16]. After each inspection, the decision maker 
should decide whether a replacement should be done to repair 
the system or the equipment could be used until the next 
inspection, and introduce the optimal structure and policy. In 
2009, a research was done by Radouane Laggounea et al 
about preventive maintenance planning for multi-component 
system with not-negligible replacement times. The impacts of 
maintenance time on the desirable policy have been shown 
clearly by numeral results in an oil refinery [17]. 

In this paper, maintenance, as well as production operation 
and the relations between them have been modeled by the 
optimal value of state and control in systems. In addition to 
differences in production states, other differences in control 
activities of this process have been considered, and control 
models have been categorized based on target function. The 
target is to minimize the total cost of maintenance controls in 
different states of production. 

III. SIGNS AND SYMBOLS OF PROBLEM 

The marks used in this paper are shown in A. and the problem 
is described in Section B. 

A. Symbols used in the paper 

S: All states of system facilities 
S t : System state at time t 

P [s t , s t+ i]: Transition probability from states s t to s t+ \ 

P: Transition matrix State S 
U: Maintenance activities 

U t : Control element of U, u 0 (no maintenance) to u n 
(replacement) 

P(u): Possibility of U control 

P S st • Change state of s to s' with the control u 

M: Matrix of the cost of production in the equipment failure 

M w : non-production equipment costs in the event of failure of 

W. 

P [Uils] : states transition probability when ui control runs. 


g u (s t ,s t+ i): Change the state of t to t + 1 the cost control when u 
apply. 

7i(s,u): Select a control policy u in state s. 
g J : The cost of policy n when state s choice 
r 00 : Return value function, the policy n is the expected 
return starting from states S. 

Q n (s f u): The control value, the expected return of the S 
mode with the control policy u is n. 

/^(s): Minimum function return values for all policies 
Q*(s f u): Minimum value for all policies is the function 
control. 

y : Discount factor and a maximum is one 
n*(_s, li): The best control policy u in state s 

B. Problem expression 

The decision support system is designed to optimize the 
maintenance and operation processes, and to integrate the 
production, taking into account the equipment failure effects 
on Markov decision process based-production volume, and 
then, the production and maintenance data are collected 
according to localization of Markov decision process. Finally, 
the optimal values of production state and of maintenance 
control are calculated, and the optimal policy is then chosen, 
using the created modern approach. Using the maintenance 
data of six years ending 2013, the model has programmed 
each group of equipment used in petrochemical companies in 
a way that the changes in production rate have the least impact 
on quality and the extent of equipment depreciation, which in 
turn increases the overall system effectiveness. The Markov 
decision process model consists of a set of different operation 
states S, a set of possible maintenance actions U, weights of 
various equipment MW, and the real cost of G(S, U) function 
[18] -[25]. 

This research tries to find the real cost of G(S, A) function 
with regard to repairing groups in the maintenance system, 
and production through modeling different combination 
scenarios and operating actions. Then, using Markov decision 
process, the optimal control policy is obtained based on the 
minimum forced cost [26] - [41]. 


IV. Problem calculations 

The defined objective of problem is to find the optimal 
control policy of maintenance for production systems, while 
the weight of the used equipment set are variable. 

In the problem, the activity s is defined as the function 7i, 
where n is the policy of decision maker for a state where S has 
been chosen. 

The objective has been to choose the policy n, when the 
accumulation function sums of stochastic costs were at their 
minimum. 

The sum of expected costs over a six-year period is defined as 

(3):_ 

£n'=iEf=i K.-Y' t 9u t S t+1 ) where u t = tt (s t ) (3) 

v is the discounting factor with a maximum of one (0<y<l); 
M w is the weight of any equipment; and g Ut (% S t+1 )is the 
cost imposed upon system when control u is performed, and 
the transition from t to t=l take places. 
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A. Production states 

The production states of system are shown as set S, which 
includes all states. 

The initial state in Markov decision process is defined as s, 
and the alternative state as s’. Then, the transition state is 
assumed as following: 

P S S’= P [$b ^t+ll 

B. Control states 

States of system maintenance (control) is defined as follows. 

U Maintenance Actions — {^0? ul , u2, u3, U 4, . . . } 


C. The cost of production of each type of equipment 
failure. 

Lost production costs of manufacturing facilities during 
different conditions in the matrix M are shown. 


'M 11 ■ 

■■ MIS' 

.MSI ■ 

■■ M55. 


D. the status of the maintenance control 

Figure (1) states the overall system different from controls are 
shown. 



Figure 1: show the different positions of the controller 


E. Calculation of the probability of changing the state by 
control of the policy n 


Policy change control condition of equation (4) is 
obtained. P g y = XueU^O* u)P^r (4) 


F. Calculation of cost control by policy k 
The cost of changing the control mode of relation (5) is 
obtained. g"= E U£ u ^(s, u)g” (5) 


G. Calculation of the state value function and the control 
value function 

State value function of a Markov Decision Process MDP 
policy 7i is the expected return starting from state S, that the 
relation (6) is obtained. 
j"(s) = tit | St = s ] (6) 

Control value function of a Markov decision process is 
expected to return to the start of the S state control policy u is 
7i. From equation (7) is obtained. 
Q^fau) - E„ [j t Ijt = s, Ut = u] (7) 


H. Bellman Expected equation 

Bellman equation for the value function state expected to 
affect the cost of lost production due to downtime of 
equipment, the relation (8) is defined. 
j"00 = [nit+igt+i + Y F (St+i) I St = s] (8) 

Bellman equation for the value function to control the impact 


of the expected cost of lost production due to downtime of 
equipment, the relation (9) is defined. 

Q^fou) = [m t+1 g t+1 + y <T( s t+ll u t+1 ) | ^ = 

S,Ut = u ] 

(9) 

Figure (2) represents the expected Bellman equation for the 
value of the control u control the state s to state s' to change its 
state when the policy 7i is chosen. 



Figure 2: Bellman equation for the expected value of the state 
s to state s’ with the control u and the choice of policy n 
Equation (10) Bellman equation for the value of state s is 
expected, when the state s to s' is done with the policy n. 

F GO = SueuTrCs.u) (M b G- + Y Ss'esPss' F (s')) 
( 10 ) 

Equation (11) is expected Bellman equation, u is the control 
value when the state s to s' is done with the policy n. 

Q™ (s, u) = 

MgGs + Y Es'esPs* EufFuTCO^u') Q™ (s', u')(ll) 
According to equation (10) the expected Bellman equation 
can be summarized as equation (12) should be used. 

r = M' n G K + Y p' n jT (12) 

After obtaining the relation (12), the expected Bellman 
equation of state value by equation (13) is obtained for the 
various policies. 

J” = (1+ y P ' n ) _1 (13) 

IV.IX. Calculate the optimal policy 

The optimal policy is the minimum control value in the 
production states obtained through (14). 

u * (s, u) = f 1 if u = argmin aeA Q* feu) 
l- 0 otherwise 

(14) 

The obtained policy is the definite optimal policy based on 
Markov decision process. 


V. Numerical computation problem 

A. Frequency of changes in year 

Frequency of changes in production conditions in Table (1) is 
displayed. 

Table 1: Cumulative Frequency Percent of Total Change 


State Change 

No 

State Change 

No 

State Change 

No 

from 0 to 0 

0 

from 40 to 0 

0 

from 75 to 0 

0 

from 0 to 40 

0 

from 40 to 40 

0 

from 75 to 40 

1 

from 0 to 75 

1 

from 40 to 75 

2 

from 75 to 75 

6 

from 0 to 100 

0 

from 40 to 100 

0 

from 75 to 100 

1 

from 0 to 120 

0 

from 40 to 120 

0 

from 75 to 120 

0 

Sum 

1 

Sum 

2 

Sum 

8 
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B. Frequency of equipment failures 
In Table (2) the number of defects in the study is shown. 


Table 2: the number of defects in equipment 


Categories 

1387 

1388 

1389 

1390 

1391 

1392 

Ordinary 

38 

189 

202 

208 

404 

389 

Important 

25 

220 

280 

467 

450 

602 

The average 

22 

201 

189 

564 

347 

386 

Important 

1 

3 

2 

6 

4 

2 

Very important 

0 

0 

0 

0 

1 

0 

unique 

0 

0 

0 

0 

0 

0 


C. The weight of each type of equipment 

The cost of production of each type of equipment downtime 
percentages imposed by the production of matrix M is shown. 


■o% 

0% 

0% 

0% 

0% - 

0% 

0% 

0% 

0% 

0% 

0% 

0% 

0% 

15% 

35% 

0% 

0% 

20% 

45% 

65% 

1-0% 

15% 

50% 

75% 

95%- 


D. change between different states of the system 
State transition matrix in Table (3) is displayed. 

Table 3: State-transition matrix 



0 

40 

75 

100 

120 

0 

0% 

0% 

0% 

0% 

0% 

40 

0% 

0% 

14% 

0% 

100% 

75 

100% 

100% 

72% 

100% 

0% 

10 

0 

0% 

0% 

14% 

0% 

0% 

12 

0 

0% 

0% 

0% 

0% 

0% 


E. change production modes with different control 

P°75,ioo=P(0)x P(75,100)=4.61% x 100%=4.61% 


Table 4: Calculate the value of control 


Q(s,u) 

u=0 

U=1 

u=2 

u=3 

s=0 

0.00000 

0.00000 

0.00000 

0.00000 

S=1 

0.08795 

0.00876 

0.00698 

0.00068 

s=2 

0.24428 

0.02433 

0.01940 

0.00189 

s=3 

0.77847 

0.07755 

0.06181 

0.00601 

s=4 

1.57948 

0.15734 

0.12542 

0.01220 


Q(s,u) 

II 

2 

u=5 

c 

II 

os 

s=0 

0.00000 

0.00000 

0.00000 

S=1 

0.00000 

0.00000 

0.00000 

s=2 

0.00000 

0.00000 

0.00000 

s=3 

0.00000 

0.00000 

0.00000 

s=4 

0.00000 

0.00000 

0.00000 


H. selection of the optimal control policy 

According to (14), and calculated that the optimal control 
policy is selected. 

So in general, for y = 1, Q (s, u3) = 679193, third time horizon 
optimal control options and policy choices are counted. 
tt* : (s, 3)=1 


VI. Conclusion 

In this paper, a model was designed to choose the optimal 
control policy provided that the activities of maintenance and 
production decisions in the context of Markov decision 
process depend on each other, and the impact of lack of 
production volume is considered, when the equipment fails in 
the production support system. 

The optimal control policy has been chosen and implemented 
using the calculations of the fifth section over a specific 
period. Then, sampling of equipment was done according to 
the studied statistical population to measure the overall 
system effectiveness where indicators of reliability, failure 
rate of equipment, mean time between two failures, and, at the 
end overall system effectiveness were calculated over a 
six-month period both before and after the implementation of 
the optimal policy. The aforementioned indicators have been 
shown in Table 5, and positive and acceptable effects could 
be seen there. 


F. Calculate the state value function 

The calculation according to equation (11) in state matrix 
valued function value is calculated with a discount rate. 


(0.0) I 

( 0 . 00069 ) 


0.22915 
0.92205 
■ 1.87444 - 


Table 5: comparing indicators before and after 
implementation of the optimal policy 


Kind of Indicator 

Before the run 

After the run 

Reliability 

78.09% 

89.99% 

Failure rate 

0.45% 

0.01% 

Mean time between 
failures 

0.64 day 

3.83 day 

OSE 

82.7% 

87% 


G. Calculate the function value control 

Control value with a discount rate according to equation (11) 
in Table (4) is displayed. 


References: 

[1] Mobery, J. (1997). Reliability - Centered Maintenance. New 
York, USA: Industrial Press Inc. 

[2] Dellagi, Sofiene, Rezg, Nidhale, Gharbi, Ali. (20 10). Optimal 
maintenance / production policy for a manufacturing system 


18 


www.erpublication.org 


International Journal of Engineering and Technical Research (IJETR) 
ISSN: 2321-0869, Volume-3, Issue-1, January 2015 


subjected to random failure and calling upon several subcontractors. 
International Journal of Management Science and Engineering 
Management. 5(4):26 1-267 

[3] Srinivas Kumar Pinjala, L. P. (2006). An empirical investigation on 
the relationship between business and maintenance strategies. 
Production Economics. 104(1):2 14-229. 

[4] Keith Mobley, R. (2002). An Introduction to Predictive Maintenance. 
2nd edition. New York, USA: Elsevier Science. 

[5] Feld, W. (1998). Lean Production: Tools, Techniques and how to use 
them. Virginia, USA : St. Lucie Press 

[6] Shum, Yu-Su, Gong, Dah-Chuan.(2010). Design of an integrated 
maintenance management system. Journal of the Chinese Institute of 
Industrial Engineers. 20(4):337-354 

[7] Guo, X.P. and Hernandez-Lerma O. (2003). Zero-Sum Games for 
Continuous -Time Markov Chains with Unbounded Transition and 
Average Payoff Rates. Journal of Applied Probability. 40: 327-345 

[8] Howard, R. A. (1960). Dynamic Programming and Markov Processes. 
London, UK: The M.I.T. Press. 

[9] Kageyama, M .(2008). On optimality gaps for fuzzification in finite 
Markov decision processes. Journal of Interdisciplinary Mathematics. 
11(1): 77-88 

[10] Mrinal K. G. and Subhamay S.(2013). Non-Stationary Semi-Markov 
Decision Processes on a Finite Horizon. Journal of Stochastic 
Analysis and Applications. 31(1) : 183-190 

[11] Puterman, M. L. (2005). Markov Decision Processes: Discrete 
Stochastic Dynamic Programming. New Jersey ,USA : Hoboken 

[12] Faddoul , R, Wassim ,R. and Chateauneuf, A. (2011). A generalised 
partially observable Markov decision process updated by decision 
trees for maintenance optimisation. Journal of Structure and 
Infrastructure Engineering. 7(10): 783-796 

[13] Koochaki, Javid B, Jos A. C, Wortmann, H, Klingenberg, 
Warse.(2012). The influence of condition -based maintenance on 
workforce planning and maintenance scheduling. International 
Journal of Production Research. 51(8):2339-2351 

[14] AlDurgam, Mohammad M, Duffuaa, Salih 0.(2012). Optimal joint 
maintenance and operation policies to maximize overall systems 
effectiveness. International journal of production research. 
51(5):1319-1330 

[15] Zhang, Zaifang,Chu, Xuening.(2010). A new approach for conceptual 
design of product and maintenance. International Journal of 
Computer Integrated Manufacturing .23(7):603-618 

[16] Murat Kurt and Jeffrey P. Kharoufeh (20 10). Optimally Maintaining a 
Markovian Deteriorating System with Limited Imperfect Repairs. 
European Journal of Operational Research, 205: 368-380. 

[17] Radouane Laggounea, Alaa Chateauneuf and Djamil Aissania(2009). 
Preventive maintenance scheduling for a multi-component system 
with non -negligible replacement time. International Journal of 
Systems Science.41(7):747-761 

[18] Anderson, W. J. (1991). Continuous -time Markov chains. Berlin 
, Germany: Springer. 

[19] Bain, L. a. (1991). Statistical analysis of reliability and life testing 
models- Theory and Methods. New York, USA: Marcel Dekker. 

[20] Bather, J. (1976). Optimal stationary policies for denumerable 
Markov chains in continuous time. Journal of Advances in Applied 
Probability. 8: 144-158. 

[21] Bellman, R. (1957). Dynamic programming. Princeton, USA : 
Princeton University Press. 

[22] Bellman, R. E. (1975). Dover paperback Dynamic Programming. 
Princeton ,USA : University Press 

[23] Bertsekas,D. P. (2001). Dynamic programming and optimal control, 
Vol. n. Belmont USA: Athena Scientific. 

[24] Beutler, F. J., & Ross, K. W. (1985). Optimal policies for controlled 
Markov chains with a constraint. Journal of Mathematical Analysis 
and Applications. 112: 236-252. 

[25] Blackwell, D. (1962). Discrete dynamic programming. Journal of 
Annals of Mathematical Statistics. 33: 719-726. 

[26] Borkar, V. S. (1991). Pitman research notes in mathematics and 
Optimal control of diffusion processes. Harlow, UK: Longman 
Scientific & Technical. 

[27] Cao, X. R. (2000). A unified approach to Markov decision problems 
and performance sensitivity analysis. Journal of automatics (Oxford). 
36: 771-774 

[28] Cao, X. R. (2003). Semi-Markov decision problems and performance 
sensitivity analysis. International Journal of IEEE Transactions on 


Automatic Control. 48: 758-769. 

[29] Cao, X. R. (2007). Stochastic learning and optimization: a 
sensitivity-based approach. Berlin, Germany: Springer. 

[30] Cavazos-Cadena,R.(1991). A counterexample on the optimality 
equation in Markov decision chains with average cost criterion. 
International Journal of Systems and Control Letters. 16: 387-392. 

[31] Chen, M. F. (2004). From Markov chains to non -equilibrium particle 
systems. Singapore: World Scientific. 

[32] Feinberg, E. A. (2004). Continuous -time jump Markov decision 
processes: a discrete-event approach. International Journal of 
Mathematics of Operations Research. 29: 492-524. 

[33] Feinberg, E. A., & Shwartz, A. (2002). Handbook of Markov decision 
processes. Dordrecht, Netherlands: Kluwer Academic. 

[34] Goldberg, D. E. (1989). Genetic Algorithm in Search, Optimization & 
Machine Learning. New York, USA: Addison -Wesely Publishing 
Company. 

[35] Huang, Chun-Chen, Yuan, John. (2011). A general maintenance 
policy for a multi-state deterioration system with multiple choices of 
imperfect maintenances. Journal of the Chinese Institute of Industrial 
Engineers. 28(5):336-345 

[36] Liu , Q , Tan, H and Guo, H.(2012). Denumerable continuous -time 
Markov decision processes with multiconstraints on average costs. 
International Journal of Systems Science. 43(3):576-585 

[37] Pearl, J. (1984). Heuristic Intelligent search strategies for computer 
problem solving. New York, USA : Addison -Wesley Publishing 
Company. 

[38] Rabbinge, D.W. (1985). Dynamic programming and the computation 
of economic injury level for crop disease control. Agricultural 
Systems. 18(4):207-226 

[39] Shih, His-Kung and Dah-Chuan Gong.(201 1). Efficient Methodology 
for Facility Location Problem Solving with Combination of Delphi 
Method, AHP, and DEA. Journal of Management. 28(1): 81-96. 

[40] Singh, N and Brumer,P.(2012). Efficient computational approach to 
the non-Markovian second order quantum master equation: electronic 
energy transfer in model photosynthetic systems. Journal of 
Molecular Physics. 110(15): 1815-1828 

[41] White, D. J. (1993). Markov Decision Processes. Chichester, UK: 
John Wiley & Sons Lid 


19 


www.erpublication.org 


